HTML5 Streaming Video Slow?

There are many factors that could be in play. Depending on the network connection and saturation of that connection ( meaning how much other data is being sent on that same connection/isp ) there can be intermittent issues where some videos will play fine because that connection is more free and unused at that particular moment, while loading another video 2 minutes later, if the connection was more saturated with other traffic, then the user would experience lag.
There are measures that can be taken to help mitigate that lag for users who may not have the best connection or who may be using a network connection that is physically very far from the data center, however these measures would involve using content delivery networks where you would be charged by the gig transferred along with storage used on that network. This would essentially be a process where a copy of the videos would reside on multiple servers that are physically closer to the end user, therefore decreasing latency. CDNs ( content delivery networks ) can be costly, especially for video content that contains many gigs of data, and they are only effective when you utilize enough distribution points so that your user base is effectively covered. They can also be complex with enforcing any kind of intellectual property protection, some CDNs do not even provide a measure to insure intellectual property protection to even verify that a user is a valid user before providing content, meaning any video on that CDN would technically be public if the user had the correct URL.