You can't stop the EF 20-35mm f/3.5-4.5 down to f/3.5. That is because f/3.5 is the f-number when the aperture is wide open and not stopped down at all. It can only open up to f/3.5 with the lens' focal length set to between 20mm and 23mm.

I'm going to take a wild guess and assume that you're trying to get the lens to open up to f/3.5 at a focal length longer than 23mm? What you seem to be trying to do is to open up to f/3.5 when the lens won't go any wider than f/4 or f/4.5 at the focal length the lens is set. Is this correct?

The short answer is that the lens can't, and never could, be used at f/3.5 at focal lengths longer than 20-23mm.

The EF 20-35mm f/3.5-4.5 is a variable aperture lens. What this means is that as the focal length increases, the lens' maximum f-number also increases. When there are two f-numbers in a lens' model name or description, the first number is for the largest aperture (lowest f-number) when the lens is used at the widest focal length. The second number is for the largest aperture (lowest f-number) when the lens is used at the longest focal length. Both numbers are the largest aperture (lowest f-number) the lens can do at the respective ends of the focal length range.

One reason your EF 20-35mm f/3.5-4.5 lens can be so lightweight and compact is because of design decisions that resulted in a variable aperture. A constant aperture 20-35mm lens that maintains f/3.5 throughout the entire zoom range would be heavier, more complex, and require all of the zooming elements to be between the front of the lens and the physical aperture diaphragm. By placing some of the zooming movement behind the aperture diaphragm, the lens can be made more compact (and cheaper). One of the compromises made in such a design is that the lens will have a variable maximum aperture.

When you remember being able to use the lens at f/3.5, you had to have had the lens zoomed to somewhere between 20mm and 23mm. At 24mm-38mm the widest aperture available is f/4. At 39mm-40mm the widest aperture available is f/4.5.

You may use narrower apertures (higher f-numbers) at any of the lens' available focal lengths. But the widest available aperture (lowest f-number) will vary depending upon the focal length to which the lens is zoomed.

For why variable aperture lenses have higher f-numbers as they are zoomed to longer focal lengths, please see the following existing questions and their answers here at Photography SE:

How do zoom lenses restrict their widest aperture at the telephoto end?

Why doesn't the picture become darker the more you zoom in?