إِنْ هِىَ إِلَّآ أَسْمَآءٌ سَمَّيْتُمُوهَآ أَنتُمْ وَءَابَآؤُكُم مَّآ أَنزَلَ ٱللَّهُ بِهَا مِن سُلْطَـٰنٍ ۚ إِن يَتَّبِعُونَ إِلَّا ٱلظَّنَّ وَمَا تَهْوَى ٱلْأَنفُسُ ۖ وَلَقَدْ جَآءَهُم مِّن رَّبِّهِمُ ٱلْهُدَىٰٓ
In hiya illaaa asmaaa'un sammaitumoohaaa antum wa aabaaa'ukum maaa anzalal laahu bihaa min sultaan; inyyattabi'oona illaz zanna wa maa tahwal anfusu wa laqad jaaa'ahum mir Rabbihimul hudaa
These are nothing but names you have invented — you and your forefathers — for which God has sent down no authority. They follow only conjecture and what their lower selves desire, even though guidance has already come to them from their Lord.
Yusuf makes this same argument from a prison cell in Egypt, telling his two companions that what they serve are names they and their fathers named, with nothing sent down to authorise them. Those are the only two places the case is put in these terms, one to the leadership of Mecca and one to two prisoners. Sulṭān here means a warrant, the kind of backing a claim would need to stand.