In the early 1800’s, the town of Brownsville was mainly populated by the Dutch. They used the land primarily for farming. The land also contained factories manufacturing stone slabs, used to create buildings. In 1858, the land was founded by William Suydam. He divided the land into 262 lots, providing shelter for workers who lived there. Only three years later Suydam ran out of money, needing to pay the mortgages. The land was then auctioned off in 1866 to Charles S. Brown, naming “Brownsville” after him.

During the 1880’s to the 1950’s, Brownsville was mainly populated by Jews. Charles S. Brown advertised the area’s open spaces to Jews who lived in Lower Manhattan. There were 250 houses in “Brown’s Village” by 1883, most of them being factory workers who commuted to Manhattan.  In 1887, Elias Kaplan described Brownsville as a more favorable town, compared to the Lower East Side. He built a factory and houses for his workers could have a place to live and work. Other manufacturers that created food, furniture, and metals were created within the next decade, settling their factories in Brownsville. This led to more housing being built there, increasing the population.

By 1900, approximately 25,000 people lived in Brownsville. Majority of people lived in two-story wooden houses built for two families. Many of these buildings were overcrowded, having up to eight families living in some of these two-family houses. Many of these houses lacked running water, and their wood construction made these houses easy to catch on fire. Around this time the population shifted toward an African-American and Latino majority. Most of the new residents were poor and socially disadvantaged, especially the new African-American residents coming from the South because of  Jim Crow laws. The neighborhood was racially segregated, both blacks and Jews lived in Brownsville. The end of the Jewish population in Brownsville happened in the 1950s, when the New York City Housing Authority decided to build more new public housings. The whites quickly moved out, reducing the population to blacks and Latinos.