No Titus is Not speaking just of pastors. Also the people that it is speaking about are to be the most Godly men in the church that are called to be setting the example for others in the church. That is how we also know that the Apostles taught against domestic violence, it is clearly against the description of what an ELDER in the church should look like/behave like. Elders are pastors, bishops, deacons, elders of all sort are given authority and responsibilities for the care of the laity.
Tit 1:5 For this cause left I thee in Crete, that thou shouldest set in order the things that are wanting, and ordain elders in every city, as I had appointed thee:
Tit 1:6 If any be blameless, the husband of one wife, having faithful children not accused of riot or unruly.
Tit 1:7 For a bishop must be blameless, as the steward of God; not selfwilled, not soon angry, not given to wine, no striker, not given to filthy lucre;
Tit 1:8 But a lover of hospitality, a lover of good men, sober, just, holy, temperate;
Tit 1:9 Holding fast the faithful word as he hath been taught, that he may be able by sound doctrine both to exhort and to convince the gainsayers.
1Ti 3:12 Let the deacons be the husbands of one wife, ruling their children and their own houses well.