Your syntax is fine. You probably have an index floating about. A common cause is when you want to *change* the definition of what the primary key is. For example:
SQL> grant dba to uat identified by uat;
Grant succeeded.
SQL> conn uat/uat
Connected.
SQL> create table uat.my_table ( x int , y int);
Table created.
SQL> create index PK on uat.my_table ( x ) ;
Index created.
SQL> alter table uat.my_table add constraint PK primary key ( x ) ;
Table altered.
SQL> alter table uat.my_table drop constraint PK;
Table altered.
SQL> alter table uat.my_table add constraint PK primary key ( x,y ) ;
alter table uat.my_table add constraint PK primary key ( x,y )
*
ERROR at line 1:
ORA-00955: name is already used by an existing object
We created an index on 'x', then when we dropped the constraint, the index was kept. When we wanted to added a new PK constraint on (x,y) we wanted to create an index called PK on (x,y) ... and we couldnt.
Hope this helps.