John H. Mercer was born in Cheltenham, England, on 29 October 1922, the third child of Harriet and John W. Mercer. In 1965, Mercer married Mary Fink, with whom he had one daughter, Jane, who was born in October 1971. During World War II he served in the British Merchant Marine (1940-1946) as a radio man. Following the war, he entered the University of Cambridge, where he received a Bachelor of Arts degree in geography in 1949. He then moved to Canada, where he received his doctorate in geography from McGill University in 1954.

Mercer was a Research Scholar at the Australian National University in Canberra from 1954 to 1956, where he studied land use and population in western Samoa. He returned to Canada and worked in the Canadian Hydrographic Office as a geographer from 1957 to 1958. The American Geographical Society employed him at its World Data Center A for Glaciology in New York (1959-1960; 1961-1962; 1964; 1966). In this capacity, he wrote the Southern Hemisphere Glacier Atlas (1967) and contributed extensively to the Atlas of Mountain Glaciers of the Northern Hemisphere (W.O. Field, editor).

In 1960, he became a Research Associate at Ohio State University's Institute of Polar Studies, later renamed the Byrd Polar and Climate Research Center. He became the Institute's first Senior Research Scientist, and remained at Ohio State University until his death. Through the University, Mercer initiated and took part in numerous field studies to further the study of glacial geology. These included Canada's Baffin Island (1952-1953); Alaska's Prince William Sound (1967); western Greenland (1968); Antarctica (1960-1961; 1964-1965; 1969-1970), notably in the Reedy Glacier and Beardmore Glacier areas of the Transantarctic Mountains; Argentine Patagonia (10 field seasons between 1963 and 1985); Chilean Channels (11 field programs between 1969 and 1987); Peruvian Andes (1974, 1976, 1977, 1981); and New Zealand. With the study of the Quaternary Andean glaciations being one of his major research focuses, Mercer's extensive field work in South America, especially concerning the "tropical" Quelccaya Ice Cap in Peru, has helped synthesize the spatial variations of global climate change (an underlying theme of his research).

His work on the West Antarctic Ice Sheet (WAIS) was also particularly influential in the study of global climate change. Mercer's paper, "Antarctic Ice and Sangamon Sea Level," published by the (IASH) in 1968, presented glacial geological evidence that the size of WAIS had changed a number of times in the past. If the shelves of ice at the rim of the WAIS were warmed, Mercer contended, the larger mass of ice might then slide into the ocean, disintegrate into icebergs, and raise sea levels. In 1978, Mercer suggested in an article in magazine that global climate change caused by use of fossil fuels could cause this disintegration and sea level to surge by more than 5 metres within 50 years.

Mercer died on July 3, 1987, in Columbus, Ohio. In 2002, Ice Stream A, the southernmost of several major ice streams draining from Marie Byrd Land into the Ross Ice Shelf, flowing west to Gould Coast to the south of Whillans Ice Stream, Antarctica, was renamed the Mercer Ice Stream in his honor.