SQL Agent Won’t Start After Switching the Service Account

My problem started shortly after I switched the service account that powered the SQL Services. On the initial install I used my domain login during the install. Then I received the good old change password nag and changed my password three or four days later. Life went on just peachy until I had to reboot. So in order to avoid this again I decided to use a different domain account. Using SQL Server Configuration Manager I changed all the services to use the new domain account and began the process of starting each service leaving the agent as the last. To my surprise I received a nice little error:

The request failed or the service did not respond in a timely fashion. Consult the event log or other applicable error logs for details.

At this point I thought a reboot might resolve this which it didn’t. So I decided to listen to the error message and search through the error logs. In the Windows Event Viewer I found the following:

Login failed for user ‘xxxxx\sqlservice’. Reason: Failed to open the explicitly specified database. [CLIENT: <local machine>]

After reading this I thought the SQL permissions were incorrect so I double checked those and they turned up fine. So I proceeded on. This time to the SQL error logs in the instance directory. In my case “C:\Program Files\Microsoft SQL Server\MSSQL10_50.DEV08R2\MSSQL\Log”.

I opened the ERRORLOG with notepad and began reviewing the contents and I found some interesting information. Essentially it stated:

Error: 18456, Severity: 14, State: 38.
Login failed for user ‘XXXXX\sqlservice’

Now I am thinking perhaps I mistyped the password which seemed like a viable action seeing the rest of the services were working fine. So I visit the Configuration Manager again and attempt to redo the password. For Schnitz and Giggles I decided to deliberately enter in the wrong password and surprisingly I received a nice error:

The specified network password is not correct.

Now I am puzzled. The password was right all along yet the previous error is saying that the login failed. WTF? To rule out if this is a Windows permissions or SQL Server permissions issue I added the service account to the local Administrators group and suddenly I was able to start the service. At the very least I now know that this is a Windows permission issue. Now I just need to figure out where this permissions issue resides.

I started to review the accounts and remembered during the installation that SQL Server creates Windows groups. So I hone in on the SQLServerMSSQLUser$XXXXXX$DEV08R2 group and find that the service account is not listed. So I add the service account to this group and kick off the Agent service and boom… it works! I tried changing the service account on a SQL Server 2005 instance and I noticed that the Windows group was updated so I am not sure why SQL Server 2008 R2 didn’t behave the same way.

The actual permissions are required within the instance directory. Here is where I found it.

And the required permissions are as illustrated.

Side Note

Once you switch the service account make sure you take a look at the owner for the databases and jobs. In my case I wanted all the databases and jobs to have the owner of the sqlservice so I ran the following query to address the database owner.

SELECT 
	name
	,'USE [' + name + ']; EXEC sp_changedbowner ''xxxxxx\sqlservice''' 'cmd'
	,SUSER_SNAME(owner_sid) 'owner'
FROM sys.databases
WHERE (SUSER_SNAME(owner_sid) != 'xxxxxx\sqlservice')
   OR (SUSER_SNAME(owner_sid) IS NULL)

Then I copied the contents from the ‘cmd’ column and executed them to make the change.


Note: you cannot change the owner of the master, model, msdb and tempdb databases. If you attempt to change the owner of the system databases you will be greeted with an error.

To circumvent this you can add an additional condition to the WHERE clause as followed and this will skip the system databases.

SELECT 
	name
	,'USE [' + name + ']; EXEC sp_changedbowner ''xxxxxx\sqlservice''' 'cmd'
	,SUSER_SNAME(owner_sid) 'owner'
FROM sys.databases
WHERE (SUSER_SNAME(owner_sid) != 'xxxxxx\sqlservice')
   OR (SUSER_SNAME(owner_sid) IS NULL)
  AND (database_id NOT IN (1,2,3,4))

The stored procedure sp_changedbowner will be removed in a future version of SQL Server. Which means you need to learn how to use ALTER AUTHORIZATION. Which I picked up from Jes’s article Changing a SQL Server Database. Here is the revised script you would use to generate the ALTER AUTHORIZATION statement.

SELECT 
	name
	,SUSER_SNAME(owner_sid) 'owner'
	,'ALTER AUTHORIZATION ON DATABASE::' + name + ' TO [sa]' 'cmd'
FROM sys.databases
WHERE (SUSER_SNAME(owner_sid) != 'xxxxxx\sqlservice')
   OR (SUSER_SNAME(owner_sid) IS NULL)

Lastly I ran the following query to generate the ‘cmd’ statement which changed the job owner for all the jobs I had set up. Keep in mind that if you have any backup jobs that are backing up to a network share you will need to update the share permissions and add the service account to permit writing to that location.

SELECT
	name
	,'EXEC msdb..sp_update_job @job_name = ''' + name + ''', ''@owner_login_name = xxxxxx\sqlservice''' 'cmd'
	,SUSER_SNAME(owner_sid) 'owner'
FROM msdb..sysjobs	
WHERE (SUSER_SNAME(owner_sid) != 'xxxxxx\sqlservice')
   OR (SUSER_SNAME(owner_sid) IS NULL)

Advertisements

Goodbye Stored Procedures… Hello Maintenance Plans

I like to write my our procedures for the sheer fact that I like the control. Which means I had sprocs that cleared out the history logs retaining only the records within the last 90 days. I also had sprocs that performed system and user database backups which would backup to a network share and append my timestamp (i.e. master_YYYYMMDD_HHMM.bak) which were all executed by agent jobs. Depending on the need I would setup Full, differential and log backups; furthermore, my process would also purge old back up files.

SELECT
'_' +
CAST(CONVERT(VARCHAR(8),GETDATE(),112) AS VARCHAR)
+ '_' +
CAST(REPLACE(CONVERT(VARCHAR(5),GETDATE(),108),':','') AS VARCHAR)
+ '.bak'

I am here to say I have given up my ways and have been walking a different path. I moved away from my custom methods and adopted Maintenance Plans. I use them for backing up the system and user databases if the system is not being serviced by a backup system as well as running DBCC CHECKDB against each database. I also use maintenance plans to purge the history logs that falls outside of a 90 day threshold and perform reorgs or rebuilds of my indexes. For a complete task list visit: Maintenance Tasks.

I am kidding, it’s hard to break away from writing my own procedures. Maintenance Plans are good if you’re getting started with SQL Server and really don’t understand the fundamentals of writing your own queries to accomplish the same outcome. I don’t know if this makes me a control freak or not. Oh wait I said that in the beginning of this post. Anyhow if you want to setup maintenance plans here is what you need to do.

One thing to know before we jump in is that you need to grant the SQL Agent Service account write access to the backup share in my case the Agent service is “sqlservice”. Typically you would use a domain account (i.e. domain\sqlservice) which you will need to grant Change access on the directory where you will be storing the backups (i.e. \\server\backups\).

If you are using the default file path which was set during the initial install of SQL Server then the path would be like <drive letter>:\..\MSSQL10_50.<instance name>\MSSQL\Backup\ and will be using the SQLServerMSSQLUser$<computer name>$<instance name> which has enough permissions.

Also you need know Maintenance Plans and the SQL Agent are not features you will find in Express editions. Well technically the SQL Agent is there, but it cannot be used/started in Express edition, so hopefully you have at the very least developer edition.

Now that I touched on a few reasons on why to use Maintenance Plans let’s kick our heels up and dig right in. You’ll need to be a sysadmin in order to create a maint plan so if you’re not then the following steps will be more informational than anything. If you have worked with SQL Server Integration Services then the Maintenance Plans design surface will look a bit familiar.

For this post we will be using two tasks: Backup Database and Maintenance Cleanup to schedule routine backups of the system database: (master, model and msdb). Once you run through this process creating another Maintenance Plan to address the user databases is extremely similar with literally one item to change. You’ll see what I am talking about in a moment. Let’s being…

Creating a Maintenance Plan for daily backups of System Databases

Step 1: Open up Management Studio, connect to your instance and expand Management
Step 2: Right click on Maintenance Plans and select New Maintenance Plan…
Step 3: Enter a specific name: (i.e. Daily_System_DB_Full_Backup) & click OK

If you are going to schedule a FULL backup let’s say on Sunday then the name of the Maintenance Plan would be along the lines of (i.e. Weekly_System_DB_Full_Backup). Then if you were going to incorporate daily differentials you might consider calling the Maintenance Plan: (i.e. Daily_System_DB_Diff_Backup). Perhaps you want to perform Full backups of the DBs daily and then perform Transaction Log backups hourly or every four hours, etc… you would want to make the names meaningful to the types of backups the Maintenance Plan is performing. Enough harping on that… I am beginning to hear crickets at his point.

Once you have clicked OK you arrive at the design surface where you will build your workflow process. The default is always Subplan_1 which you can leave or you can change the name. If your Maintenance Plan only utilizes a single subplan then the naming really doesn’t matter, but if you add additional subplans then you might want to consider naming them accordingly to better identify their purpose or intent as opposed to having Subplan_2, 3, etc… you get my drift.

To rename the subplan just double-click on the Subplan name.

Rename the Subplan and specify a description and simply click OK

Since I am going to use only one subplan for this post I am going to leave that name as Subplan_1.

Step 4: Drag the Back Up Database Task to the designer surface area

Now in the Toolbox pane to the left select and drag Back Up Database Task from within the Maintenance Plans Tasks to the designer surface area and release.

Step 5: Edit the task

Right click on the Back Up Database Task and select Edit…

Step 6: Perform a quick hat trick (hockey reference)

A) Click the Database(s) select list
B) Select System databases
C) Click OK

Step 7: Specify the backup path in the Folder: field and click OK

If the Folder: is already populated using a local file system path then you can leave it as is. It will work as is. I am using a network share just as an example to illustrate that you would normally want to backup to a network share which should be routinely backed up via an enterprise backup system.

Step 8: Drag the Maintenance Cleanup Task to the designer surface area and release. The Maintenance Cleanup Task will purge all files with a specific file extension using a given date specification.

Step 9: Set the workflow

A) Click the Back Up Database Task
B) Drag the Green Arrow to the Maintenance Cleanup Task and release

This means when the backup task completes successfully to proceed and execute the Maintenance Cleanup Task.

Step 10: Edit the Maintenance Cleanup Task

This is similar to Step 5 except you are right clicking on Maintenance Cleanup Task instead and selcting Edit…

Step 11: Four point play…

A) Specify the backup path in the Folder: field
B) Specify the extension bak in the File extention field
C) Specify a value unit of time in numbers
D) Specify a value unit of time for the time frame then click OK

Step 12: Scheduling the Maintenance Plan

Click the Calendar Icon just above the Maintenance Plan name

Step 13: Secondary hat trick…

A) I like to remove the .Subplan_1 from the Maintenance Plan name (optional)
B) Click the occurs: select list and choose Daily
C) Set the field occurs once at: to when you want this to kick off and click OK

Step 14: Close & save the Maintenance Plan

Click Yes to confirm

Success!

Done! Well at least with creating the Maint Plan. We just need to test it.

Step 15: Start the newly created SQL Job

Right click on Daily_System_Database_Full_Backup.Subplan_1 and select Start Job at Step…

If the process fails make sure your service account has Change permissions on the network share. If you are running the SQL Agent Service under the Network Service account then set the permissions as illustrated. This also applies to what ever account you are running the Agent Service under like a domain account and such.

Otherwise if everything is configured correctly you will see GREEN!

And the databases will have been backed up as such…

Amended 2011-05-10


You should read Brad McGehee’s eBook: Brad’s Sure Guide to SQL Server Maintenance Plans to get a better understanding of some of the gotchas you need to be aware about in regards to Maintenance Plans.