Button Click Delay inside ScrollView

I have an android activity that takes up more then the full screen so I place it inside a ScrollView. There is a button at the very bottom of my Activity and when I scroll down there is a delay in which I can click the button. The delay is until the scrollbar fades away and then the开发者_运维问答 button can be clicked.

This is not a very long time, but for a user it may seem like it and it is very annoying so is there a way to make the buttons clickable while still scrolling or make that time shorter at least?