Born to Lithuanian Jewish parents, Levine grew up in the South End of Boston, where he observed a street life composed of European immigrants and a prevalence of poverty and societal ills, subjects which would inform his work. He first studied drawing with Harold K. Zimmerman from 1924-1931. At Harvard University from 1929 to 1933, Levine and classmate Hyman Bloom studied with Denman Ross. As an adolescent, Levine was already, by his own account, "a formidable draftsman". In 1932 Ross included Levine's drawings in an exhibition at the Fogg Art Museum at Harvard, and three years later bequeathed twenty drawings by Levine to the museum's collection.Levine's early work was most influenced by Bloom, Chaim Soutine, Georges Rouault, and Oskar Kokoschka.Along with Bloom and Karl Zerbe, he became associated with the style known as Boston Expressionism.
From 1942 to 1945 Levine served in the Army. Upon his discharge from service he painted Welcome Home, a lampoon of the arrogance of military power; years later the painting would engender political controversy when it was included in a show of art in Moscow, and along with works by other American artists, raised suspicions in the House Un-American Activities Committee of pro-Communist sympathies. In 1946 he married the painter Ruth Gikow and moved to New York City.