There has to be a way! Making Linked Servers to MS Access faster

For a few days now I have been bashing my head on a performance problem in SQL Server 2005, although I think it would apply to any version of SQL Server.
The issue is that in order to process the data I am working with I need to rely on a Microsoft Access database located on a different Windows Server. To get to this data I am using a Linked Server that is configured with the Microsoft.Jet.OLEDB.4.0 provider. The only configured provider options are “Allow inprocess = True”.

Getting the data I need is not the problem… getting it FAST is the problem.

From my Googling and Bing’ing I’ve found that tweaking the provider and/or the Linked Server options I can get different performance compared to the defaults… still not good (or even better) performance, just different.

What I found was that changing the “Collation Compatible” and “Use Remote Collation” to TRUE:
the first 5 times the stored procedure (that uses the linked server connection) is called… it executes immediately – really fast retrieval from the linked server. All subsequent calls take about 10 seconds round trip.

WHY?

I’ve tried using OPENQUERY, OPENROWSET, four-part naming – with default options on the linked server and without. I just cannot get the linked server to retrieve data quickly every time the sproc is called.

To work around the problem I started caching the data from the Microsoft Access database locally on SQL Server (updated hourly via a job). Not ideal since real-time data is preferred, but adequate for now.

So my question to all the SQL Server / Microsoft Access experts out there is:

Why is it so slow, what environmental factors can be tweaked to get better performance?