What is the acceptable similarity in a mathematics PhD dissertation when checking by Turnitin?
Is this percentage acceptable by most committees?
This is the wrong question to be asking, since academic decisions are not made based on a numerical measure of similarity from a computer program. The purpose of this software is to flag suspicious cases for humans to examine more carefully. It will identify passages that appear similar to other writings, but it can't decide whether that constitutes plagiarism.
For example, part of your thesis might be based on previous papers you have written. In some circumstances, it may be reasonable to copy text from these papers. (You need to check that your advisor approves and that it doesn't conflict with any university regulations or the publishing agreement with the publisher.) Of course you would need to cite the papers and clearly indicate the overlap. It's not plagiarism if you do that, but Turnitin doesn't understand what you've written well enough to distinguish it from plagiarism. So it's possible that Turnitin would flag lots of suspicious sections, but that your committee would look at them and see that everything is cited appropriately.
If you haven't committed any plagiarism, then you don't need to worry about this at all. If you genuinely write everything yourself (or carefully quote and cite anything you didn't write), then there's no way you
could accidentally write something that looks like proof of plagiarism. There's just too much possible variation, and the probability of matching someone else's words by chance is negligible. The worst case scenario is that Turnitin flags something due to algorithmic limitations or a poor underlying model, but human review shows that it is not actually worrisome. (Nobody trusts Turnitin more than they trust their own judgment.)
I'll assume you don't know you've committed plagiarism, but it is possible that you honestly wouldn't know? Unfortunately, the answer is yes if you have certain bad writing habits. For example, it's dangerous to write while having another reference open in front of you to compare with. Even if you don't copy anything verbatim, it's easy to write something that's just an adaptation of the original source (maybe rewording sentences or rearranging things slightly, but clearly based on the original).
If that's what worries you, then you should take a look at the most suspicious passages found by Turnitin. If they look like an adaptation of another source, then it's worth rewriting them. If they don't, then maybe Turnitin is worrying you unnecessarily.
But in any case a plagiarism finding won't just come down to a percentage of similarity. Any percentage greater than 0 is too much for actual plagiarism, and no percentage is too high if it reflects limitations of the software rather than actual plagiarism.