Oliver grew up in Philadelphia, Pennsylvania, playing soccer with the Lighthouse Boys Club, winning the 1948 and 1949 U-19 National Championships. He also played for the Kensington Blue Bells. Oliver attended Northeast High School in Philadelphia. He also played for the Philadelphia Nationals of the American Soccer League in 1946-1947, scoring thirteen goals in eighteen games. He attended college at Temple University, playing on the school’s soccer team from 1951 to 1954. In 1951, Oliver was selected as a second team All American as the Temple claimed to be the national champion. Oliver was selected as a first team All American and Temple was named the national champion. Oliver was selected for the second year in a row as first team All American in 1954, his senior season.
After graduating from Temple, Oliver began playing as an amateur with Uhrik Truckers in the American Soccer League. The Truckers took the 1955 and 1956 ASL championships, but Oliver broke his leg during the 1955-1956 season. In 1958, he was drafted into the U.S. Army. Being stationed in Ludlow, Massachusetts, he played half a season with the Ludlow Lusitano, the local ASL team. He was then transferred to San Francisco, California where he played for the amateur San Francisco Mercury, an ethnic Russian team. Following his discharge from the Army in 1959, he signed with Baltimore Pompei. Oliver was a 1960 All Star, but Pompeii folded in 1961. He did play for the U.S. soccer team which finished 0-4 at the Football at the 1963 Pan American Games.
In his later life, Oliver also became an USSF ’A’ license coach and referee, and regularly trains new coaches in the Washington, DC area. He was inducted into the Philadelphia Old Timers Hall of Fame in 1998 and the Virginia-DC Soccer Hall of Fame in 2001