In January, 1940 Sam meets Alice Neel at a WPA meeting. They start an affair that continues on and off until 1955. A year later their son Hartley Neel is born. In December, 1941, Sam and Claire divorce.
In 1942, his relationship with Neel having soured, he moves to Washington state and begins a stint with Kaiser Shipyards working on Liberty ships.
He is joined by his brother Lou, who's wife Sylvia had recently died of tuberculosis.
Soon after, Sam moves to Los Angeles unsuccessfully seeking a reconciliation with Claire. He finds work in the film industry, working as a French language dubber for MGM (where he came to know Yul Brynner), and making a 16mm color documentary on Betta Fish for a major studio.
Near the end of World War II, Sam returns to New York City. He restarts a complicated, uncertain, on and off relationship with Neel.
In 1946 Sam completes Of These Our People, a short film on the legacy of anti-semitism. He acts as director, writer and cameraman.
During the rest of the 1940's and through the 1950's he works various jobs as a camera operator, photographer and photofinisher including a stint with Fordel Films, a producer of medical films and other documentaries.